Incremental Learning for RNNs: How Does it Affect Performance and Hidden Unit Activation?
نویسندگان
چکیده
In this short paper we summarise our work [5] on training first-order recurrent neural networks (RNNs) on the abc language prediction task. We highlight the differences between incremental and non-incremental learning – with respect to success rate, generalisation performance, and characteristics of hidden unit activation.
منابع مشابه
On the effect of low-quality node observation on learning over incremental adaptive networks
In this paper, we study the impact of low-quality node on the performance of incremental least mean square (ILMS) adaptive networks. Adaptive networks involve many nodes with adaptation and learning capabilities. Low-quality mode in the performance of a node in a practical sensor network is modeled by the observation of pure noise (its observation noise) that leads to an unreliable measurement....
متن کاملImproving Deep Learning by Inverse Square Root Linear Units (ISRLUs)
We introduce the “inverse square root linear unit” (ISRLU) to speed up learning in deep neural networks. ISRLU has better performance than ELU but has many of the same benefits. ISRLU and ELU have similar curves and characteristics. Both have negative values, allowing them to push mean unit activation closer to zero, and bring the normal gradient closer to the unit natural gradient, ensuring a ...
متن کاملSeismic Performance of seat-type bridges with elastomeric bearings
In Iran and some other countries, elastomer bearings in seat-type bridges are used with no sole/masonry plates and there is no positive connection between superstructure and substructure. Different codes have diverse provisions regarding the coefficient of friction (μ) between elastomer bearing and superstructure/substructure and also the design strength of shear keys (Vsk). Developing a finite...
متن کاملOutlier Detection Using Extreme Learning Machines Based on Quantum Fuzzy C-Means
One of the most important concerns of a data miner is always to have accurate and error-free data. Data that does not contain human errors and whose records are full and contain correct data. In this paper, a new learning model based on an extreme learning machine neural network is proposed for outlier detection. The function of neural networks depends on various parameters such as the structur...
متن کاملModeling and Analyzing Incremental Quantity Discounts in Transportation Costs for a Joint Economic Lot Sizing Problem
Joint economic lot sizing (JELS) addresses integrated inventory models in a supply chain. Most of the studies in this field either do not consider the role of the transportation cost in their analysis or consider transportation cost as a fixed part of the ordering costs. In this article, a model is developed to analyze an incremental quantity discount in transportation cost. Appropriate equatio...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2003